Towards a Neurocognitive Model of Turntaking in Multimodal Dialog

Communicating individuals take turns speaking, gesturing, and interacting according to their goals and circumstances. The process is referred to as turn taking and it is a major organizing mechanism in real-time dialog. Thorough psychological studies have shown there to be a significant amount of similarity between societies with regard to observed behaviors during conversation (O’Connell et al. 1990). More recently, turn taking has become an issue in robot and virtual human research as researchers aim to make these systems more fluent and dynamic when interacting naturally with humans (cf. Gratch et al. 2006; Leβmann et al. 2004; Maxwell et al. 2001; Lemon et al. 2001; Bischoff 2000; Thórisson 1993). In spite of an apparent simplicity (what could be simpler than taking turns in speaking?), turn taking is a complex phenomenon that has eluded clear definition to date, although Sacks et al. (1974), Goodwin (1981), and others have certainly made measurable progress in that respect. Multimodal natural communication involves many perceptual, planning, and motor mechanisms. A comprehensive model of turn taking must address not only how people produce hierarchically organized actions but also how they recognize these when produced by others. One path towards building a biologically plausible, inclusive model of cognitive mechanisms involved in real-time turn taking is by integrating models at different levels of description, for example cognitive and neural. A key assumption in the present work is that turn taking during conversation exists primarily (although certainly not solely1) for the purpose of helping participants reduce cognitive load during the interpretation of the content of the conversation. Any production related to the topic of the dialog naturally interferes with the interpretation process; therefore, understanding deteriorates when we try to speak at the same time as we try to listen. Because of this, perception of behaviors that are not related directly to dialog content but rather have the goal of guiding the overall interaction, such as back channel feedback (Yngve 1971) and